Abstract: In this talk, we present two major challenges in peer review, propose solutions with guarantees, and discuss important open problems.
(1) Fraud: There have been several recent discoveries of fraud in peer review: A group of participants form a coalition, get assigned each other's papers by manipulating the system, and then accept each others' papers. We present an algorithm which mitigates such fraud by randomizing reviewer assignments, and does not rely on assumptions about the malicious behavior. The algorithm yields an optimal-quality assignment subject to randomization constraints, and we will also discuss experiments characterizing this tradeoff.
(2) Feedback: Real-world systems rely on feedback about their performance for their continual improvement. A useful means of obtaining feedback about the peer-review process is to ask authors' opinions. However, author opinions are significantly biased by whether their paper was accepted. We formulate this problem and present algorithms, with statistical guarantees, to debias such feedback. Our work relies on the key observation that the direction of this bias is known: the program chairs know which authors' papers were accepted.
An overview of challenges and computational solutions in peer review is available here: http://bit.ly/PeerReviewOverview